Rubrics as Rewards
RL with verifiable rewards (RLVR) works in math and code because either they have clear answers or their outcomes are verifiable. However, most real-world tasks have no verifiable answers. To handle this gap, RaR assigns a weighted checklist of criteria to each prompt, and then use LLM judger to score the response w.r.t. the criteria list for GRPO reward computation.
Table of Contents
1. Background
2. Methodology
2.1. Reward Formulation
Rubric Modeling. RaR formulates the question as follows. Each prompt \(x\) in the dataset has \(k\) corresponding rubrics, represented by a set of pairs
\[ \set{(w_{j}, c_{j})} \]
Here, \(w_{j}\) is the weight of rubric \(j\); \(c_{j}\) is a predicate function, usually a LLM judger for non-verifiable tasks, that accepts the prompt \(x\) and model’s response \(\hat{y}\) and produces a single value \(0,1\) to tell if the response satisfies the rubric, i.e.,
\[ c_{j}(x, \hat{y}) \in \{ 0,1 \} \]
Reward Formulation. To construct rewards from rubrics, RaR proposes 3 variants.
- Explicit Aggregation
- The rubric score are weighted-averaged. The final reward is \[ \frac{\sum_{j} w_{j} c_{j}}{\sum_{j} w_{j}} \]
- Implicit Aggregation
- An LLM will be utilized to give a final reward in 1-10 scale (which is then normalized to 0-1) by inspecting the prompt, the response, and all rubric items.
- RLVR as Special Case
- Setting \(k=1, w=1, c=\text{exact match}\) degrades RaR to RLVR.
2.2. Rubrics Generation
The rubrics generation should follow 4 properties:
- Rubrics are generated with reference answer grounding.